1. Field of the Invention
The present invention relates to a communication system and communication method thereof, with PCI-Express used in an interface, and with a plurality of nodes connected through a PCI-Express.
2. Description of the Related Art
Peripheral Component Interconnect (PCI) is a standard for computer bus connecting between each part inside computers, and is currently applied to most of computers; however, specification of PCI-Express, which is faster transfer interface standards, is newly developed.
For example, as described in Patent Document 1, in the PCI-Express, in order to prevent skews between signal lines, which occur in a parallel bus, a serial link is applied, and packets are transferred a synchronously through the serial link. Also, the PCI-Express allows upgrade in units of layers, and therefore functions thereof are layered. By adding a header in a transaction layer to transmission data produced in number and a CRC are added to the head and the tail of the transaction packet, generating a data link packet. Finally, in the physical layer, it is converted to physical signals and transmitted through a transmission media.
Meanwhile, in the PCI-Express, between nodes can be connected by a switch (PCI-Express switch).
FIG. 1 is a diagram explaining the conventional PCI-Express communication system with each node (also referred to as a module herein) connected by the PCI-Express switch performing communication between modules A-n based on the PCI-Express specification.
In the following description, a part of the PCI-Express relating to the present invention is further explained.
According to the PCI-Express specification, a module, called a root complex, which is located on top of the upper layer, such as a module A(1) in FIG. 1 is required. Also, as shown in FIG. 1, each module has been assigned a bus number (BusNum.) and a device number (DeviceNum.) in each port of the PCI-Express switch (6). FIG. 1 shows an example that the module A(1) has the bus number of 0 and the device number of 1 assigned, a module B (2) has the bus number of 1 and the device number of 1 assigned, a module C(3) has the bus number of 1 and the device number of 3 assigned, and a module D (4) has the bus number of 1 and the device number of 4.
Next, packet communications between modules via the PCI-Express switch (6) is explained.
There are two types of packet routing on top of the PCI-Express switch (6); address routing for routing according to an address comprised in a packet header, and ID routing for routing according to a requester ID constituted from device and action numbers, comprised in a packet header.
These two types of routing, in the PCI-Express specification, are separately used so that address routing is used for a request packet, and the ID routing is used for a response packet. Therefore, the address routing is used for a request packet for a memory write request command and a memory read request command. Note that a response by a response packet is not made to the memory write request command in this PCI-Express specification.
As shown in FIG. 1, (1) when a memory read request command is issued from the module D(4) and the module C(3), (2) the PCI-Express switch (6) transfers a memory read request packet to the module C(3) according to the address routing of the PCI-Express specification, (3) the module C(3) responds read data as a response packet. (4) The PCI-Express switch (6) transfers the response packet to the module D(4) according to the ID routing of the PCI-Express specification.
FIG. 2 is a flow diagram explaining a GET transfer sequence, which reads out data from another end. Firmware in transmitting side starts up hardware in the transmitting side by generating a descriptor, which is for specifying data to be read out, in the memory. As shown in FIG. 2, transmitting side or transmitting source issues a memory read request packet as a GET transfer sequence, and transmits the GET transfer request to the receiving side according to address routing of the PCI-Express specification in the PCI-Express switch (6). In the receiving side, hardware in the receiving side starts up from the idle state by a receive-trigger of the receiving transmitted from the transmitting side. And after reading out data from the memory, a response to the GET request is returned as a response packet. The response packet is transmitted back to the transmitting side according to ID routing of the PCI-Express specification of the PCI-Express switch (6). There hardware in the transmitting side receives read data from the response packet, interrupts the transmitting side firmware by writing the data in the memory, and the transmitting side firmware carries out termination processing.
FIG. 3 is a flow diagram explaining sequence of PUT transfer, which writes data at the other end. Routing of the memory write request, packet issued from the transmitting side is performed according to address routing of the PCI-Express specification as in the case of a GET transfer. And the receiving side does not make a response according to the PCI-Express specification (response is not generated in the non-posted memory write). For that reason, it was impossible for a requesting side to determine whether the write processing normally performed.
For example, assume a bus number and a device number are set in each port of the PCI-Express switch (6) as described in FIG. 1, and a memory read request packet as a GET transfer performed according to address routing of the PCI-Express specification. The module C transfers read data as a response packet, and the response transfer control is performed according to ID routing of the PCI-Express specification.
FIG. 4 shows a part of a header in the transaction layer of a packet used in the above sequence. A requester ID in FIG. 4 is information for identifying a module that transmits the packet and an address in FIG. 4 contains routing information for routing to a transfer destination. In the address of the response packet, the requester ID of the request packet corresponding to the response packet is set, and the PCI-Express switch (6) performs routing by using a bus number of the set requester ID when it is in the ID routing of the response packet and performs routing in accordance with the part in which the address in FIG. 4 is contained in the address range set in each port when it is in the address routing.
As shown in FIG. 5, with such a system, when the module A(1), which is the root complex, is separated, for example, due to an abnormality, such as abnormal voltage, chip defect, software problem, or maintenance, etc., because of the absence of the root complex, according to the PCI-Express specification any of other modules B-N is required to become a new root complex and a change is required in component information (bus numbers/device numbers) of each port that the PCI-Express switch (6) possesses. In an example shown in FIG. 5, after module A(1) separation, the module B(2) is the new route complex with bus-number 0 and device-number 1.
In FIG. 5, suppose (1) a memory read request is issued from the module D(4) to the module C(3), (2) the PCI-Express switch (6) transfers a memory read request packet to the module C(3) according to address routing of the PCI-Express specification, and (3) at that time, the module A(1), which is a root complex, is separated due to abnormality. Then, (4) configure information of each port in the PCI-Express switch (6) is changed, the bus number and the device number are replaced (it may be referred to as ID replacement hereinafter), and the module B(2) becomes the new root complex. (5) The module C(3) responds read data by the memory read request command from the module D(4), and the requester ID at that time, comprises the bus number and the device number of the module D before the replacement. (6) The PCI-Express switch (6), because the packet transmitted from the module C(3) is a response packet, performs routing according to ID routing of the PCI-Express specification. (7) However, because the bus number and the device number are assigned to other modules due to the replacement, the response packet may be transmitted to wrong modules by mistake, or may be lost.
[Patent Document 1]
Japanese Published Patent Application No. 2004-326151